Skip to content

Expand AlbertAlert source catalog with additional CT incident/prevention/context feeds - #111

Merged
potemkin666 merged 26 commits into
mainfrom
copilot/update-news-feed-for-relevance
Apr 24, 2026
Merged

Expand AlbertAlert source catalog with additional CT incident/prevention/context feeds#111
potemkin666 merged 26 commits into
mainfrom
copilot/update-news-feed-for-relevance

Conversation

Copilot AI commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

The source expansion request combined high-velocity narrative feeds with official operational/prevention sources to improve terrorism signal coverage across UK, EU, and international lanes. This update adds the missing endpoints while avoiding duplicates already present in the catalog.

  • Scope added (new source-of-truth entries)

    • Added 23 new entries across shard files:
      • data/sources/international/context.json
      • data/sources/uk/context.json
      • data/sources/uk/incidents.json
      • data/sources/uk/prevention.json
      • data/sources/eu/context.json
    • New additions include SouthFront, IntelTechniques, Reddit r/worldnews / r/unitedkingdom, Urban Terrorism in Europe, IPT, AllAfrica Terrorism, IntelNews, ICT, INTERPOL news page, ProtectUK, MI5 terrorism news category, CPS terrorism-relevant news surface, Home Office Terrorism Act stats collection, NPSA threats, Dutch OM terrorism pages, and German Federal Prosecutor EN releases.
  • Deduplication and lane placement

    • Skipped endpoints already covered in catalog (e.g., ZeroHedge, War on the Rocks, Bellingcat, Long War Journal, GNET, EU Home Affairs, ICCT baseline).
    • Placed sources into appropriate lanes (incidents, prevention, context) with keyword-gating flags aligned to source reliability/noise profile.
  • Catalog compilation

    • Regenerated compiled catalog output in data/sources.json from updated shard files.

Example of added entry pattern:

{
  "id": "protectuk-news",
  "provider": "ProtectUK - News",
  "endpoint": "https://www.protectuk.police.uk/news",
  "kind": "html",
  "lane": "prevention",
  "region": "uk",
  "isTrustedOfficial": true,
  "requiresKeywordMatch": true
}

Copilot AI and others added 25 commits April 24, 2026 11:19
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/e35a7380-f184-4b48-b5a2-eeba227f5f66

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/a6ef3658-00ee-4153-bbb0-89b02f29c930

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/ac11d610-cf28-4a85-9bbc-64ceca78b17d

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/d1bd349c-abf8-4a5e-a0f9-45a40f6136d8

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/f89e3cc3-0892-4518-ad2e-51a83f06a9dd

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/2763ec1c-1bb8-4bc8-a411-5404054fe40b

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/eb6f7b92-625a-4530-874b-ced5e26e6717

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/745ab5f6-1f03-4d4f-84fb-30e2027a42c3

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/745ab5f6-1f03-4d4f-84fb-30e2027a42c3

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/5c1ddfa5-618b-438a-b7cb-be6e1863d911

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
Agent-Logs-Url: https://github.com/potemkin666/AlbertAlert/sessions/5c1ddfa5-618b-438a-b7cb-be6e1863d911

Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>
@potemkin666

Copy link
Copy Markdown
Owner

@copilot resolve the merge conflicts in this pull request

@potemkin666
potemkin666 marked this pull request as ready for review April 24, 2026 18:08
Copilot AI review requested due to automatic review settings April 24, 2026 18:08
Co-authored-by: potemkin666 <183807833+potemkin666@users.noreply.github.com>

Copilot AI commented Apr 24, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Resolved. I merged origin/main into this branch, resolved all merge conflicts, and pushed commit b39db4b.

Copilot AI requested a review from potemkin666 April 24, 2026 18:11
@potemkin666
potemkin666 merged commit bd70b44 into main Apr 24, 2026
1 check failed
@potemkin666
potemkin666 deleted the copilot/update-news-feed-for-relevance branch April 24, 2026 18:13

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1158bac9bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"provider": "Europol a Media and Press",
"endpoint": "https://www.europol.europa.eu/media-press",
"kind": "html",
"endpoint": "https://www.europol.europa.eu/cms/api/rss/news",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep source endpoint unique to avoid silent source drops

This endpoint now duplicates europol-news-rss in the same shard, and this commit introduces several similar endpoint collisions; scripts/validate-feed-data.mjs rejects these duplicates, and at runtime normaliseSourcesPayload() drops later duplicate endpoints (scripts/build-live-feed/io.mjs), so intended catalog additions are silently ignored. Please keep each source endpoint unique before recompiling data/sources.json.

Useful? React with 👍 / 👎.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR primarily expands and refreshes the AlbertAlert source catalog across multiple regions/lanes (adding new feeds and replacing stale/noisy endpoints), and it also updates live-feed operational behavior around quarantine decisions and stored-alert retention.

Changes:

  • Expanded/updated source-of-truth shard catalogs (UK/EU/Europe/International/London/US) with many new endpoints and several replacements (HTML→RSS/Atom where available).
  • Changed auto-quarantine + CI quarantine seeding logic to rely on consecutive-failure thresholds (not health-score thresholds), and updated related tests + quarantine review artifacts.
  • Added a per-source/per-day cap when selecting stored alerts, and adjusted freshness sorting.

Reviewed changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
tests/seed-quarantine.test.mjs Updates expectations for quarantine seeding now based on failure thresholds rather than low health score.
tests/error-pattern-analysis.test.mjs Adjusts tests to reflect that never-verified sources are no longer immediately quarantined on first definitive failure.
tests/decision-logic.test.mjs Adds tests for per-source/day stored-alert cap and freshness sorting; updates imports accordingly.
source-quarantine.html Refreshes generated snapshot metadata and adjusts generatedAt fallback.
shared/alert-view-model.mjs Changes tie-breaker ordering so newer publishedAt wins within the same freshness bucket.
scripts/ci/seed-quarantine-from-health.mjs Removes health-score threshold logic from quarantine seeding; uses consecutive failures + existing quarantine flag.
scripts/build-live-feed/config.mjs Updates hard-skip list (re-enables Jamestown Terrorism Monitor id).
scripts/build-live-feed/alerts.mjs Enforces a cap of 2 stored alerts per source per day when selecting stored alerts.
scripts/build-live-feed.mjs Revises auto-quarantine logic to trigger only after repeated consecutive failures; updates quarantine review inclusion logic.
data/top-20-source-remediation.json Regenerated remediation output (now empty in this snapshot).
data/source-remediation-sweep.json Regenerated remediation sweep output (now empty in this snapshot).
data/sources/us/context.json Updates several endpoints (e.g., CTC/RAND) to RSS/alternate pages and adjusts comments.
data/sources/uk/prevention.json Adds several UK prevention sources and upgrades some endpoints to RSS.
data/sources/uk/oversight.json Adds UK Parliament RSS index source.
data/sources/uk/incidents.json Adds multiple UK incidents sources and upgrades some endpoints/comments.
data/sources/uk/context.json Updates many UK context endpoints and adds numerous new ones (including replacements).
data/sources/london/incidents.json Updates an entry to use Met “latest news” listing and adjusts metadata.
data/sources/london/context.json Swaps/updates several London context endpoints and adds new local RSS feeds.
data/sources/international/prevention.json Refines Tech Against Terrorism entry to a more focused page.
data/sources/international/oversight.json Tightens keyword-gating on UNOCT oversight source.
data/sources/international/context.json Adds many international context feeds (including Jamestown TM and various RSS sources).
data/sources/europe/prevention.json Updates ISD endpoint and removes stale query parameter from a RUSI replacement.
data/sources/europe/incidents.json Updates comment text for Portugal crime source entry.
data/sources/europe/context.json Switches some context sources to RSS and replaces FT world with AP terrorism hub.
data/sources/eu/prevention.json Adds several Germany-focused prevention sources (CeMAS/KN:IX).
data/sources/eu/oversight.json Replaces FRA endpoints with other oversight sources and updates HRW surface.
data/sources/eu/incidents.json Adds many EU incidents sources (notably German federal/state police + related).
data/sources/eu/context.json Replaces/updates multiple EU context endpoints (ECFR/Eurojust/Europol/Supo/Statewatch etc.) and adds several new ones.
data/sources/eu/border.json Updates comment for Frontex archive entry.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +349 to 352
// Auto-quarantine only after repeated consecutive failures.
const overFailureThreshold = next.consecutiveFailures >= AUTO_QUARANTINE_FAILURE_THRESHOLD;
if (overFailureThreshold && !next.quarantined) {
next.quarantined = true;

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After changing quarantine logic away from health-score review thresholds, HEALTH_SCORE_REVIEW_THRESHOLD is no longer referenced in this file (it only appears in the config import list). If linting checks for unused imports, this will fail CI; please remove the unused import/export from the import destructuring (or reintroduce a use).

Copilot uses AI. Check for mistakes.
Comment on lines 298 to +306
"id": "fra-rss",
"provider": "Human Rights Watch - News RSS",
"endpoint": "https://www.hrw.org/rss/news",
"kind": "rss",
"provider": "Human Rights Watch - Terrorism and Counterterrorism",
"endpoint": "https://www.hrw.org/topic/terrorism-counterterrorism",
"kind": "html",
"lane": "oversight",
"region": "eu",
"isTrustedOfficial": false,
"requiresKeywordMatch": true,
"comment": "Next-best replacement for blocked FRA feed using HRW's active human-rights news RSS."
"comment": "Direct HRW terrorism and counterterrorism topic page for focused rights oversight."

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The id "fra-rss" is now used for a Human Rights Watch page. Reusing ids across different organisations will mix health history and makes the compiled catalog misleading. Please use an HRW-specific id (and keep FRA-related ids for FRA sources).

Copilot uses AI. Check for mistakes.
Comment on lines 102 to +110
"id": "rand-terrorism-topic",
"provider": "RAND - Terrorism Topic Updates",
"endpoint": "https://www.rand.org/topics/terrorism.html",
"kind": "html",
"endpoint": "https://www.rand.org/pubs/new.xml",
"kind": "rss",
"lane": "context",
"region": "us",
"isTrustedOfficial": false,
"requiresKeywordMatch": true,
"comment": "RAND topic pages are HTML; using the terrorism topic index and relying on keyword filtering."
"comment": "Official RAND new publications RSS feed replacing the topic HTML page."

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The id "rand-terrorism-topic" now points to RAND's general new-publications feed (new.xml). That mismatch will make future maintenance/triage confusing, and may conflict with expectations of a terrorism-scoped RAND feed. Consider renaming the id/provider/comment to reflect the broader feed or switching to a RAND terrorism-specific feed if available.

Copilot uses AI. Check for mistakes.
Comment on lines +3080 to 3095
const isQuarantined = Boolean(priorEntry?.quarantined)
&& priorConsecutiveFailures >= AUTO_QUARANTINE_FAILURE_THRESHOLD;
const deferredNextFetchAt = deferred.until
|| (isLowHealth ? priorEntry?.nextFetchAt : null)
|| (isQuarantined ? priorEntry?.nextFetchAt : null)
|| sourceScheduleNextFetchAt(source, buildDate, priorEntry, true);
nextSourceHealth[source.id] = {
...(priorEntry || {}),
provider: source.provider,
lane: source.lane,
kind: source.kind,
healthScore: priorHealthScore,
quarantined: isLowHealth,
quarantined: isQuarantined,
quarantinedAt: priorEntry?.quarantinedAt || null,
quarantineReason: isLowHealth
? clean(priorEntry?.quarantineReason || 'Low health score – needs review')
quarantineReason: isQuarantined
? clean(priorEntry?.quarantineReason || 'Consecutive failures exceeded threshold')
: (priorEntry?.quarantineReason || null),

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the deferred-source health update, when isQuarantined is false the code still carries forward quarantinedAt and quarantineReason from the prior entry. That can create inconsistent states (quarantined=false but quarantinedAt/reason present), which can confuse quarantine review/audit output. Consider nulling quarantinedAt/quarantineReason whenever isQuarantined is false in this branch.

Copilot uses AI. Check for mistakes.
Comment on lines +349 to +361
// Auto-quarantine only after repeated consecutive failures.
const overFailureThreshold = next.consecutiveFailures >= AUTO_QUARANTINE_FAILURE_THRESHOLD;
if (overFailureThreshold && !next.quarantined) {
next.quarantined = true;
next.quarantinedAt = generatedAt;
next.quarantineReason = analyseErrorPattern(next.recentErrors, {
healthScore: next.healthScore,
kind: source?.kind
});
} else if (!overFailureThreshold) {
next.quarantined = false;
next.quarantinedAt = null;
next.quarantineReason = null;

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR title/description focuses on source-catalog expansion, but this PR also changes runtime behavior (auto-quarantine rules) and retention selection (per-source/day cap) in the builder. Please update the PR description to reflect these behavior changes (or split them into a separate PR) so reviewers/operators understand the operational impact.

Copilot uses AI. Check for mistakes.
Comment on lines 26 to +34
"id": "city-of-london-police-newsroom",
"provider": "Metropolitan Police Service - News",
"endpoint": "https://news.met.police.uk/",
"provider": "Metropolitan Police Service - Latest News",
"endpoint": "https://news.met.police.uk/latest_news?source=city-of-london-police-newsroom",
"kind": "html",
"lane": "incidents",
"region": "london",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "Best-effort replacement for blocked City of London Police newsroom using active Met Police London policing updates."
"comment": "Canonical Met newsroom latest-news listing."

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entry’s id ("city-of-london-police-newsroom") now points at the Met Police latest-news listing, but the catalog already has Met-specific sources (e.g., met-police-latest-news). Reusing a City-of-London-branded id for a different provider/endpoint will mix health history and makes the catalog misleading; consider either renaming this id to a Met-specific one or removing it in favour of the existing Met source.

Copilot uses AI. Check for mistakes.
Comment on lines 1117 to +1125
"id": "fmcg-gurus",
"provider": "FMCG Gurus",
"endpoint": "https://fmcggurus.com/feed/",
"kind": "rss",
"provider": "Europol - Newsroom",
"endpoint": "https://www.europol.europa.eu/media-press/newsroom",
"kind": "html",
"lane": "context",
"region": "uk",
"isTrustedOfficial": false,
"requiresKeywordMatch": true,
"comment": "Lifestyle/travel RSS from Free RSS Feed Catalog; keyword match required to avoid noise."
"comment": "Replacement for unrelated FMCG source using Europol's official newsroom archive."

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entry reuses the existing id "fmcg-gurus" for a completely different source (Europol). Source IDs are used as stable keys (health tracking, quarantine, restore UI), so reassigning an id will mix historical health/restore state and makes the catalog misleading. Create a new Europol-specific id (and likely place it in the EU region where Europol sources already exist) instead of repurposing this id.

Copilot uses AI. Check for mistakes.
Comment on lines 1293 to +1301
"id": "silly-old-sod",
"provider": "Silly Old Sod",
"endpoint": "https://www.sillyoldsod.com/feed/",
"kind": "rss",
"provider": "Combating Terrorism Center at West Point - CTC Sentinel",
"endpoint": "https://ctc.westpoint.edu/ctc-sentinel/",
"kind": "html",
"lane": "context",
"region": "uk",
"isTrustedOfficial": false,
"requiresKeywordMatch": true,
"comment": "Lifestyle/travel RSS from Free RSS Feed Catalog; keyword match required to avoid noise."
"comment": "Replacement for personal-blog source using CTC Sentinel analysis archive."

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entry reuses the id "silly-old-sod" for CTC Sentinel. Reassigning an existing id to a different provider/endpoint will pollute health history and makes it hard to reason about quarantines/restores. Please introduce a new id for CTC Sentinel (or reference the existing CTC entry in the appropriate region) rather than repurposing this one.

Copilot uses AI. Check for mistakes.
Comment on lines 1315 to +1323
"id": "sks-business-services",
"provider": "SKS Business Services",
"endpoint": "https://www.sksbusinessservices.com/feed/",
"kind": "rss",
"provider": "UN Office of Counter-Terrorism - News",
"endpoint": "https://www.un.org/counterterrorism/en/news",
"kind": "html",
"lane": "context",
"region": "uk",
"isTrustedOfficial": false,
"requiresKeywordMatch": true,
"comment": "Lifestyle/travel RSS from Free RSS Feed Catalog; keyword match required to avoid noise."
"comment": "Replacement for unrelated business feed using UN counter-terrorism news updates."

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entry reuses the id "sks-business-services" for UNOCT news. Source ids should remain stable per real-world source; repurposing them will mix unrelated health/quarantine data and creates confusing restore/audit trails. Please mint a new UNOCT-specific id (and consider moving it to an international shard if appropriate) instead of reusing this id.

Copilot uses AI. Check for mistakes.
Comment on lines 168 to +176
"id": "eu-fra-rss",
"provider": "EU Agency for Fundamental Rights (FRA) a RSS",
"endpoint": "https://fra.europa.eu/en/content/rss",
"provider": "OHCHR - Terrorism and Violent Extremism",
"endpoint": "https://www.ohchr.org/en/topic/terrorism-and-violent-extremism",
"kind": "html",
"lane": "oversight",
"region": "eu",
"isTrustedOfficial": true,
"requiresKeywordMatch": true,
"comment": "FRA provides RSS options from this page; using the RSS landing page when direct feed URLs vary by topic."
"comment": "Replacement for FRA anti-bot RSS endpoint using OHCHR's terrorism and violent extremism topic page."

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The id "eu-fra-rss" now points to an OHCHR topic page. Reusing an id that encodes a different organisation (FRA) makes health tracking and catalog maintenance error-prone. Please rename this source id/provider to match OHCHR (or keep FRA under the FRA id and add OHCHR as a separate entry).

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants